Do the areas overlap? An implementation of the Permutation Test on the Overlapping Index

Perugini, A., Calignano, G., Nucci, M., Finos, L., Pastore, M.

 Cognitive Science Arena 13 - 16 October 2025 | Brixen

Groups/Conditions

\(t\)-test


    Two Sample t-test

data:  x and y
t = 0.9, df = 58, p-value = 0.4
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.30  3.35
sample estimates:
mean of x mean of y 
     10.2       9.2

Welch test


    Welch Two Sample t-test

data:  x and y
t = 0.9, df = 35, p-value = 0.4
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.34  3.39
sample estimates:
mean of x mean of y 
     10.2       9.2

But..

The Overlapping Index

The overlapping (\(\eta\)) is an index of effect size and varies from 0 and 1, formally defined in the following way:

\[\begin{eqnarray} \eta (A,B) = \int_{\mathbb{R}^n} min [f_A (x),f_B (x)] dx \end{eqnarray}\]

\(\eta (A,B)\) is normalized to one and when the distributions of A and B do not have points in common, meaning that \(f_A (x)\) and \(f_B (x)\) are disjoint, \(\eta (A,B) = 0\)

Permutation tests

  • Rearranges the data \(B\) times
  • Calculates the statistic for each new dataset of permuted data
  • Calculates \(p\)- value as the probability of obtaining equal or more extreme values
\[\begin{eqnarray} p=\frac{(\sum_{b=1}^B |t_b|\geq |t|)+1}{B+1} \end{eqnarray}\]

where \(B\) is the number of random permutations, \(t\) is the \(t\)-statistic computed on the observed data, \(t_b\) are those computed on the permuted data.

Applied to the Overlapping Index


Usually \(H_0\) is defined as the absence of an effect.

As \(\eta\) is the area of overlap, the higher the overlap, the closer the value to 1.

For simplicity, instead of working with \(\eta\) now we will work with the complement \(\zeta\), defined as \(1 - \eta = \zeta\) and define \(H_0 : \zeta = 0\) meaning that there is no difference between the densities of the two populations.

Applied to the Overlapping


The permutation test is applied to \(\zeta\):

\[\begin{eqnarray*} p = \frac{(\# \hat{\zeta}_b \geq \hat{\zeta})+1 }{B+1} \end{eqnarray*}\]

The principle is the same, we shuffle the data and calculate \(\zeta\) B times to calculate the \(p\)-value.

Back to our example

Panel [A] shows the two distributions with \(\eta = .46\) and \(\zeta = .54\), panel [B] shows the distribution of the \(\zeta\) statistic obtained through permutation (\(p < .001\))

The simulation study

We simulated from the Skew-Normal distribution.

The simulation study

  • \(\delta\) = (0, 0.2, 0.5, 0.8); mean of the second population;
  • \(\sigma\) = (1, 2, 3); standard deviation of the second population;
  • \(\alpha\) = (0, 2, 10); degree of asymmetry (skewness) of the second population;
  • \(n\) = (10, 20, 50, 100, 500); sample size, equal in the two samples.

For each of the 4 \(\times\) 3 \(\times\) 3 \(\times\) 5 \(= 180\) conditions we generated 3000 sets of data on which we performed the analysis.

Statistical tests for comparison

For each combination \(\delta\) \(\times\) \(\sigma\) \(\times\) \(\alpha\) \(\times\) \(n\), on the generated data were performed the following tests:

  • Permutation test on the complement of the Overlapping Index (\(\zeta_{ov}\)), \(\zeta = 1-\eta\);
  • \(F\) test (F) of homogeneity of variances;
  • Kolmogorov-Smirnov test (ks) comparing the cumulative distributions.
  • Wilcoxon-Mann-Whitney test (wmw) on ranks;
  • Welch test (w) assuming normality but not homogeneity of variance;
  • \(t\) test (t) for independent samples, assuming equal variance and normality;

Type I error: \(H_0\) True

Power: \(H_0\) False

Discussion

  • The \(\zeta\) overlapping test is the most powerful choice when assumptions are not met, also with small samples.

  • As a non-parametric test, it allows to make inference on the entire distribution without making any assumptions.

  • It is an intuitive index and it allows to investigate group differences beyond the mean .

Take home message

  • Be aware of parametric tests assumptions

  • Always plot your data BEFORE running the analysis

  • Thinking Before Testing (cit. Richard McElreath )

To find out more:

Here you can find our pre-print, on the implementation of the permutation test for the overlapping index:

https://osf.io/preprints/osf/8h4fe_v1

ambra.perugini@phd.unipd.it